Evaluating Use of Interfaces for Visual Query Specification

We present an empirical study of how users express information needs visually through “Query by sketch” interfaces. The objective of the study has been to gain an understanding of how users translate information needs into visual queries (VQ) and what type of problems they encounter. We have studied the process of translating an information need to a visual query and how a visual query interface assists or hinders the user during this process. The intent is to use this information to both improve user support in the task of formulating visual queries and assist the retrieval system in understanding the user’s query. 14 participants were given a set of image retrieval tasks and were asked to perform them in a basic VQ interface. A total of 121 visual queries were performed. The process of formulating these queries was monitored using observation, video and audio logging, and the participants were interviewed after performing the retrieval tasks. We have identified three main categories of tools for visual query formulation, freehand sketching, use of predefined shapes and use of colour. The overall use of these categories for visual query formulation has been determined. The resulting query images have been analyzed, and are categorized as either realistic or iconic, and as complete images or images with a single object of interest, providing 4 distinct categories of visual queries. We discuss the consequences of these results with regards to interfaces for visual query specification. The results indicate that the way users prefer to express information needs visually differ from the input expected by current CBIR systems. Some possible additions to QBS interfaces are suggested that might help users express their information needs as visual queries.